Skip to content

csnode: a backfill turn ends after 50 ms; slow turns are logged - #140

Merged
EnzinoBB merged 3 commits into
masterfrom
fix/backfill-turn-budget
Oct 31, 2026
Merged

EnzinoBB merged 3 commits into
masterfrom
fix/backfill-turn-budget

Conversation

@EnzinoBB

Copy link
Copy Markdown
Collaborator

Why

The mainnet test (full rocksdb node from the snapshot, master e0d2d31) backfills at ~500-600 blocks/s, but its longest processor-thread turn grew to 1744 ms for at most 50 blocks (~85 ms on average), while a mainnet round lasts ~1.2 s. On a confidant that holds consensus. Only the maximum was logged, so neither the cause nor the frequency was visible. RocksDB logged no write stalls; the log gaps at whole minutes are network-wide (the pruned node shows them too).

Change

  • Settings::turnBudgetMs (50 ms): a turn ends once it has used its time, after at least one block, besides backfill_store_per_turn.
  • A turn over 200 ms is logged (first 20, then every 100th): blocks, transactions, time in verify / store / confirmations. Stats::slowTurns; the progress line adds the count.
  • Unit test ASlowTurnStopsAtItsTimeBudget (a store hook that sleeps); the other tests run without a time limit.
  • docs/storage.md, CHANGES.

Test plan

  • CI build + unit tests (Linux, Windows, macOS)
  • robustness backfill rows (backfill-trusted-load measures the longest turn)
  • the mainnet test node runs this build

🤖 Generated with Claude Code

EnzinoBB and others added 3 commits October 31, 2026 12:22
On mainnet a full node from the snapshot (rocksdb) had backfill turns of
up to 1.7 s for 50 blocks, about 85 ms on average, while a round lasts
about 1.2 s: a confidant would hold the processor thread through a
round. A turn now also ends once 50 ms have passed (after at least one
block); a turn over 200 ms is logged with its blocks, transactions and
the time spent verifying, storing and checking confirmations, so that
the cause shows, and the progress line counts them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- a rejected block's peer key is copied before reject() erases the
  buffer entry it lives in (it was read after being freed)
- the budget and the slow-turn decision use the hooks' clock, so the
  test drives them without real sleeps
- the storage flush is timed in its log line and in the stats
- [storage] backfill_turn_budget_ms (50, 0 = no limit, at most 1000)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant